Deletion and Removal of Image Occlusion Errors 
Inventor: Alexander Berestov 



Background of the Invention 



Field of the Invention 




This invention is related to triage preceding techniques. More specifically, this 

eadatole medium for removing stereo 



\ I 

invention is a system, method, and computer* 
matching errors caused by image ocdusjj^ 



Related Application 




The subject matter of this applicatro^n^elated to the subject matter of a 
commonly owned U.S. patent applicati^n^Serial Number 09/428,286, attorney docket 
number 3457, titled "Fast Epibolar Lih^Adjustmenrof Stereo Pairs," filed on October 
27, 1999, also by Alexander Berastov' the contents ofwhich are incorporated by 



reference as if fully disclosed herein 



Related Art 





A stereo matching algorithm, whicj can produce a dense a(Td sharp depth map 
as an active range finder, is aVy technology for many stereo applications including 



atio 



object recognition, three-dimensional object modeling, vehicle navigation, and 



>n, and 
image of ; 



geometric inspection. In order to generate a three-dimensional map or image of a 
scene, however, different images of the scene must be matched properly. One of the 
maj0r Problems ,o Properiy « P0int s h th e ima9 es - wh en occ^ 

contours coincide. Occluding contours coincide when a point, which is visible in the 
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right imago, is not visible in the left image and therefore does not really have a 
matching point Alternatively, occluding errors can also occur at the borders or edges of 
an object that are\captured by a camera facing the object at different angles (called 
"occluding boundaries"). This is caused by the traditional correspondence procedure 
which will be described m further detaihbelow 




The most standard sit&^^n ^here|occlud mg contours occur is when other 
objects in the scene block the point of interest * When this occurs, area-based matching 



algorithms often give wrong disaaritWestir 



near the contour. When the classical 
search is made in the left image, the 



stereo correlation technique is applied and thej 

\ 

contour usually "leaks" to the rigilLc^he oojg^t boundary as illustrated in Fig. 2. 
Another set of errors are showfHn the top left corner of Fig. 2 and are associated with 



out-of-focus objects that cannot be matched/correctly. The present 
each of these problems, f ) 1^ 





invention resolves 



The prior art solutions used to successfully detect occlusions and avoid false 



correspondence reqwlfe^tfi^^ In the simples^case, several cameras 



may be used to capture an image of the scene from equal angles^along a hemisphere 

\ \ 

that surrounds the scene. Thus, if a point lis not included in the second image, the first 



image may be matched to tfc^tN^ and used to "complete" the occluded area in 
the second image. If not positionecTproperly, however, multiple camera stereos can 
increase the area of occlusion and may still lead to false correspondence. Mores 
specifically, depth maps generated from a polynocular stereo image often have blu\ed 
object shapes caused by the false correspondence at occluding boundaries. 
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Anoth^set of solutions involved creative manipulation of a matching algorithm. 
Some matching algorithms may be better at avoiding false correspondence problems, 
but none solves theWoblem completely. For example, feature-based matching 
algorithms, which try toscorrespond points only at object edges, may be used to avoid 
occlusion to an extent. Otner binocular stereo algorithms have also been adapted to try 
to detect "half-occluded" regions in ord^o^ii^p^ve the correspondence search. In 



both cases, however, the algorith{ns/ail-tq<neasure the depth in these regions. More 



recently, new algorithms were developed 

\ 

provide better results in occlusion detei 




Itiple camera devices, which may 



In each prior art solution, ejtt^multi^e cameras are needed to prevent 

occluded regions or the method is-extcgi^y tifne intensive and, in both cases, the 

1 \ 

resulting correspondence errors prevent creation of a complete depth map of the 

-c^^ i \ 

scene. Using multiple cameras increasesth^ cost, bur^den and complexity of the 
imaging system, and the resultiriglm&ges are still not amenable to depth analysis. What 
is needed, then, is a new methCd for dete\ing and eliminating occlusions and out-of- 

f an accurate deptn\map of the scene 
without requiring significant time and effort to accomplish 



focus errors thereby enablingJhe 




Disclosure of Invention 





A system and method for detecting and eliminating correspondence errors 



associated with image occlusions is provided. In a first embodiment of the invention, the 
method first applies traditional correspondence methods for matching points in twe 
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images, a left image 1A and a right image 1B, taken of the same scene. Ideally, the 
initial search is performed by matching each point (310) in the right image 1B with a 
"best match" point 1^20) in the left image 1 A. Once an initial set of matching points 
(310, 320) is generate^, a second search is performed by using the best match point 
(320) in the right image 1I3as the basisfor^anadditional correspondence search in the 



left image 1 A. While the firsnsearch waslferformed without restriction, the second 

i i / 

tarting doint/c 

V / 

>d. Thejpoint 



search is explicitly limited by theSstarting 



"best match" point (330) is generatec 



310) used in the first search. A second 
(330) generated in the second search 



may be the same point (310) that was\sed in the first search or may be a different 
point altogether. This results in a second set of points that represents the most accurate 



match between points. 



As will be further described beldvowith reference to Figure 3, limiting on the 
second search results fronrUhe W&y in which occlusions manifest themselves as errors 

( \r \ 

during correspondence. More specifically, incorrectly matched points often cause 

\ I \ 

leakage in a particulacdirectipn depepding on the direction <^f image used in the first 
search. If the initial points used iff 



e first search are points in the right image 1B being 

i \ 

matched to the "leff^image 1 A, then the first search will generate g^od matches for 
points on the left edgetrf objects in thd image, with a poor match on the right edge of 
the object. In this scenario, rhQsecojra search will generate good matches for points on 
the right edge of any objects in the image. By placing the additional limitations on the 
second correspondence search, the poor match points on the left side of the object will 
be avoided while still picking up the correctly selected correspondence points on the 
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right edge of rtae object. This limitation also speeds up the correspondence process 
significantly, as otaly a portion of the points in the row is used during the 
correspondence seatah. Thus, the best points from each of the searches are used to 
establish correspondence in the fastest possible fashion. 




In a second embodiment ofthe invention, the restrictions placed on the second 
search are removed and ttejiesuljing points used to accurately identify the occluded 
areas. These results mayLe used^J conjunction with the results ofthe first embodiment 
to generate an error ma6 that aeedrateiy identifies potentially problematic areas. More 

V J \ 

specifically, the results of correspondence^earch in the second embodiment avoid any 
"false positives" and can be used to further modify the results ofthe first embodiment. 

^ - \ 

Steps for removing any additional errors \n\he final images are also provided. 

j \ 



could be broken down into separate images for each 



For example, each ste 

color coordinate. The\corresponjdence search could be fqn on each image separately 
with the results uselH^rea^ disparity map for\ach color coordinate. 

Brief Description ofthe Drawings 

Figures 1 A ansM B comprise a pair of images taken of a scenevthe includes a 
can and staple, and will be used to demonstrate the present invention; 




Figure 2 is a disparity map ofthe scene that was created by combining^Figs. 1A 
and 1B using classical stereo matching algorithms; 
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5 



\ 

Figure 3 is a block diagram that illustrates the stereo matching method of the 



present invention; 




Figures 4A, 4B and 4C are individual color disparity maps that were generated 

\ / 
using the first embodiment^ the ^t^reo^ of the present invention; 

Figures 5A, 5B and 5Gyare individual color disparity maps that were generated 

v — ^ 

using the second embodiment ofLthe stereS matching method of the present invention; 



Figure 6A is a disparity mapq^eajed by combining Figures 4A, 4B and 4C into a 
single image; 



Figure 6B is the disparityMii^p of 5AHhat has been further improved by applying 
the additional error correction steps provided the present invention; 





/ \ 

Figure 7A is a disparity map created by combining Figures 5A, 5B and 5C into a 



single image; 



Figure 7B is the disparity/map of 7A that has been^further improved by applying 

/^NA 7 \ 

the additional error correctiQn^steps provided by the presenkinvention, and 

' / \ 

Figure\8 is a block diagrajn of a computer system that m^y be used to implement 
the present inv 





Detailed Description of the Preferred Embodiments 

The present invention can be used in conjunction with any numberurf different 
visual capture devices including video cameras, video capture devices on p\rsonal 

6 
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computers, standaV^photographic cameras, specialty stereo imaging cameras, or 
digital cameras. The present specification describes the invention as being used in 



conjunction with standard, photographic cameras, for illustration purposes only. 

\ 

The standard still-frame camera is normally used to capture an image of a scene 
or object. When the picture is t^ei^K ^ of the scene is "flattened" from 

three dimensions to two dimensions resultingjin the loss of information, such as spatial 

rig o 



size and the spatial relations a 
information is to take two or 
called stereo images, and to extrapol 
combine the images properlyShowever 



the 



portio 



n the image. One way of replacing the lost 
the same object from different angles, 
^spatial information accordingly. In order to 
of the first image 1A must be 



corresponded to the relev^nf^ the second image 1B. Once the images 1A, 1B 

have been matched, the imggejnformation may be u^ed to calculate the epipolar 
geometry by extracting the essential iriatrix of the system. The three-dimensional 
Euclidean structure of the imaged scene can then be comp\ted in the form of a 
disparity map or otherWee-dimensional representation. 

In the id^al case, the epjpolar lines of the two images are horizontal. In order to 




guarantee /horizontal epipolar lines, however, it is necessary to set the optical axes of 



the two cameras in parallel. For instance, calibration can be used to guarantee that the 
optical axes are parallel, the base line is horizontal, the sensors, which are\jsed to 
create the image, cbi«cider / and that the cameras have the same lens distortion. If any 
of these factors are incorrectly calibrated, however, the points in the first image iX may 
not have matching points lying along the same row in the second image 1B. \ 
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Matching points in one image 1A with points in another image 1B where both 



images are takerv^f a single scene, called the correspondence problem, remains one of 
the bottlenecks in cd^nputer vision and is important to continued development in this 

field. As is more fully described in the related applipafion entitled "Fast Epipolar 

\ 

Adjustment of Stereo Pairs? (identified above)/an adjustment algorithm can be used to 
adjust the points in the right inraage 1B in oraer to more easily correspond to the points 
in the left image 1 A, so that the^ints|n the right image 1 B are located along the same 
line as in the left image 1 A, therels^Meatiefg images with the desired epipolar geometry. 
For simplicity, we shall assun^lf^theJ 1 A,ab already have the desired 
geometry or that the adjustWnt algoriwm^as already been run on the images. 

As an initial matter, the rollowrng is a description of the classical stereo search 

\ 

that is used to establish match candidates in onaembodiment of the present invention. 
While this matching algorithmjjvilljpe used to illustrate one embodiment, those skilled in 
the art will realize that othermatching ^algorithms may ajso be used to implement the 
present invention. ' * 



centered on that-sdected phqrt iSj 
indow wilr 



As an initial matte^, for a selected point in a first imagev a correlation window 

created. The correlation window may be of any size, 

but a larger window wilryield less] precise results than a smaller window. The value of 
one or more properties of the area within the correlation window are tn$n calculated. 
For example, the mat^hi^^ may use the amount of red values i\the points 

within the correlation window as the relevant correlation property. 

An equally sized correlation window is then centered on one or more points 

within a search window in the second image. The values of one or more correlatior 

8 
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properties of arc^s within the correlation window of the second image are then 
calculated. Each point within the search window in the second image is given a 
correlation score basejJ on its similarity to the properties of the correlation window of the 
first image. 

A constraint on the correlation score can then be applied in order to select the 
most consistent matches: for^ given couple of points to be considered as a match 



candidate, the correlatiorrfcc^re 
Using the correlation technique, 
points in the second imag^ and 
matching ambiguities but^gr^L 



higher than a given threshold, for example, 
i the first image may be paired to several 
Vic& Versa. Several techniques exist for resolving the 
licity, the points with the highest correlation score will 



be selected. In the preferred embodiment of the invention, this method will be applied to 
select the best match candidates Although the^Jescribed algorithm is the preferred 
technique for matching^^ the imagesNother matching algorithms may also be 

used including: correlatiorvba^ed matching, MRF-oased matching, feature-based 
matching, and phase^based matching. 

Unfortunately\this classical correlation technique^gives greater disparity values 
in areas close to the object boundaries, making boundary a(eas difficult to analyze. This 
is particularly (true wherf using aWnaller correlation window, a\the colors in the window 
change dramatically when the border of the object is included in\he search. Using 
larger correlation windows results in less precise results, however,Vieaning that all 
points in the image, not just object boundaries, may be improperly matched. 
Referring now to Figures 1 A and 1 B, a pair of images taken of a scene 

(hereinafter referred to as the "stereo pair") including a can and stapler, is shown. This 

9 
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stereo paiijM A, 1 B will be used throughout this description to illustrate the invention. 
This illustration is not meant to limit the scope of the invention. Any number of images 
may be used, and the number or size of the objects in the scene are inconsequential to 
the operation of theteystem and method. 



Referring now to Figure 2, a green color disparity map is shown. A green color 

disparity map is gejierated By examining only the green color in two images of a scene, 

using the resulting "greerTvSfn^ A to establish a correspondence between the images, 

and using Euclidean^eomet^ to q^termine the relative distance between a point in the 

scene and the location of the imafcje capture device, such as a digital camera, used to 

\ 

capture the scene. ,m this example, the green disparity map was generated using the 

Two other: disparity maps, red and blue, (not 
shown) were also obtain M-fcorfi thejmages. Each of the disparity maps has the same 

\ 

features: occlusion erro^(^0 to the right of the objects and out-of-focus errors (220). 
These errors are generate^%y<tbe traditional correspondence algorithms. This is 
caused by the fact th^vh^griatGhing points in the left imdfcje with points in the right 



stereo pair from Fig 




image, the contours 
hand, when matching p 




leak" to the left of the objecfchoundary. On the other 
right image with one or more pc^pts in the left image, 
" to thW right of the object boundary. The present 



the contours of the^b|^s^leak ] 
invention uses these facts advantageobsly to prevent the leakage 



Referring no\AKp Figure 3, a diagram illustrating the operation of the method of 
the present invention is shoWf^JDie^method starts by running the classical stereo" 
correlation search described above. The correlation search uses a point (310) in the 



10 
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riqht^i 




ght image and searches for a best match in the left image. Once a best match point 
(320) is obtained in the left image, this point (320) is used as the basis for a second 
search that attempts to locate a best match for that point in the right image. In order to 
prevent leakage and to speed up the matching process, however, the correspondence 
algorithm does not attempt to march any points that are to the right of the original point 
(310) in the rig ht ^rhTaQ^N^h^^ea rch may result in a second best match point (330) that 
is not equivalent to the original point (310). This process results in two point pairs for 
every point in the rigl^mc^ a^rst pair of points (310, 320) that represents the best 
match between the ong^rpoint (3^0) in the right image with a matching point (320) in 
the left image, hereinafter referred to as\the "left pair"; and a second pair of points (320, 
330) that represents thesQiatpn between th\matching point (320) located in the left 
image with a point (32(6) in the right image, hereinafter referred to as the "right pair". 

/ 



In a first embodiment ofthe present inventior\each of the resulting right pairs 
(320, 330) is used taestabli§b-a correspondence between the left and right image. The 
resulting correspondence between the images may then bemused to create a stereo 



image or a disparityjnap using 



traditional Euclidean geometry?* 



The resulting red, green and blue color disparity map that were 



This method was applied (to the stereo pair 1A, 1B and used tq correspond 
points in the irpages, 

generated using Ithe right pair 020, 330) of matching points are provided inVigures 4A, 
4B, and 4C, respect ivelyTVVhile there are still errors in the correspondence (sigWfied by 
the black patches in the images), the image boundaries (410, 420, 430) are sharpen 
and can be determined with greater accuracy. Figure 4B does contain numerous errore 
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at the boundary (420), but errors can be modified or corrected. As explained with 

\ 

respect to Figure 1 above, the classical correlation technique gives greater disparity in 

V 

areas close to the object boundaries, making boundary areas difficult to analyze. Thus, 

\ 

unmodified application of the stereo correspondence process results in incorrectly 

\ y 

matched pointsHr^ning that ttje'user or process does not know that it needs to correct 
the errors. Knowledge of potentially erroneous areas, however, is a significant 
advantage in many visiorL^p||lications where the erroneous areas can simply be 

X u 

avoided as potentially representing an object boundary. This method helps resolve that 
problem. 



In sum, this rnethod combines tile best features of both searches. The left pair 
(310, 320) gives the incsyrert/results for tne right boundary of the object search, 
resulting in a value wlilchis greater than the tixie value. When we search back in the 



right image, however>^he right pair (320, 300) piteks the correct smaller disparity value, 
resulting in a proper match lafcthye right boundary. On the other hand, the left pair (310, 
320) gives the correct results foofne left boundary of the object. Although using the right 



pair (320,330) wbuld normally 
the second searches limited by 
therefore effectivel^j^s the 
results of the second search pick 



esult in greater disparity aVj improper correspondence, 
the true smaller value from tne first search, and 
brrect results as established by fhe left pair. Thus, the 
the best point pairs at both boundaries. 



Additional embodiments of this invention can provide even better^ results. In a 
second embodimeht^an additional step may be added to the first embodiment in order 
to help locate and remove errors in the correspondence. In this embodiment\f the 
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search in tlwight image locates a different match point (330) than the original point 

(310), both points are considered erroneous and are replaced with a black point. 
\ 

Therefore, only the^isparity values generated from both searches are considered 
accurate. As illustrated by the black portions in Figures 5A, 5B, and 5C, this 

\ / 

embodiment results in ^ e ^ r c onep/po\nt values but provides more certainty for those 
s^co^ In i 



values that it does 



other words, the second embodiment is the 



preferred method for the elimination of false targets. The values generated by the 
second embodiment/canlfteo b^filtered or used in combination with the results of the 

.v n \ 

first embodiment to imgroye 1 thp results. 

\ 

One benefit of this second embodiment is that it provides greater confidence in 

\ / \ 

the accuracy of the poiiqts^ir^ image. This^an be of particular use in those 
applications where pote^ia|i)MJ^known areas dan simply be avoided. In machine vision 



applications such as robotics, for example, a moving robot can simply avoid black areas 

/ 



as potentially corresponding^ am object that may prevent movement. This may lead to 



a more circuitous rptJte^Mtj^better calculated choice^vhen potentially expensive 
equipment is involved 

The rest 

the first embodiment to create a disparity confidence map. For eximple, the brightness 





nd embodiment could also be used in combination with 



of a particular pixel iMhe disparity} map can be adjusted based on the^second 
algorithm. All points in imag^4Ayfhat correspond to potential erroneous \oints in image 
5A could have a lower brightness value while "correct" points have a highePsbrightness 
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value. Other applications and embodiments could also be developed using this error 
detection step. 



A third embodimenMhat includes additional error correction steps is also 
provided. As illustrated in FigWes 4A, 4B, and 4C, each image may be split into a 

separate disparity map corresponding to eadi color coordinate. In this case, the images 

\ 

4A, 4B, 4C represent the v Red, Green, BJwe components disparity maps, respectively. 

While these color components are standard image components, other color coordinate 

systems may also be used. Regardless of the type of color coordinates used, the first 

i \ 

e di^parit^maps that corresponds to each of the 
...lue that r^iatches the value of a point at that 
same location in at least orfg3jfeth$ other disparity\maps is selected. Any point value 
that is represented by only one color kt a particular location is eliminated from the map. 



step is to compare each p^int in th 
colors. Next, every point thai 



Points in the disparity maps 



, color at _ , 
)s whida-have di 




different valuessacross every color are flagged. 



Once all erroneous areas have been identified, the errors may be eliminated 
using a simple filter or interpolati^^tg|chniques. One alternative for eliminating 
erroneous points in dlf^ar^^a^^freated using the first embodiment is to replace the 
erroneous value with the first accurate value to the right of the error \oint. In this 



alternative, the true valuers select© 
common for the c'ontourto leak to the 



Referring n 
embodiment of the 




rom the right of the error point because it is 
right of the occlusion boundary. 



ombined disparity map created using thevfirst 
inventio/i is shown. This map 6A was generated by 
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combining Figures 4A, 4B and 4C without further modification. The results of applying 
the additional error elimination steps on the disparity map 6A, i.e., replacing erroneous 
entries with a tn^ first true value to the right of that point, are shown in Fig. 6B. As 
Figure 6B demonsb^tes, the additional error filtering steps help create a vastly superior 
image resulting in a?N^tremely^ccurate depiction of the can and stapler scene. The 
small remaining errors in F(gurfe 6B can be eliminated by applying well-known filtering 
or feature analysis al§1wtT^s\ 

^ V 

Referring now to Figujey7i*v^ combined disparity map (600) generated using a 
combination of the firsfan^tHe second embodiment of the invention is shown. This 
map (600) was generated by combining Figures 5A, 5B, and 5C without further 



modification. The results 



iddih 



lying the additional error elimination steps on the 



tries wi 



disparity map 7A, i.e.[ replacing erroneous entries with the first true value to the right of 
that point, is shown in Fi"g x 7B. As Figure 7B further demonstrates, the additional steps 
help create a vastly superior image resulting in an extremely accurate depiction of the 



can and stapler scene. 



despite the greater number of error points in Figure 



7A, the final result lool^s better than in Fig. 6B because false targets were detected 
more precisely during the correspondence search. 




The additional error elimination steps are good for correcting both s 
correspondence proW^sulting from occlusions and out-of-focus errorskln areas 
that are out of focusl it is difficult to calculate exact disparity because there are^jo exact 
boundaries and the objfe^ts are blurred. \rf these cases, using either the first true v^lue 
to the right of the error or inter^&tefeRg^i value using correct points to the left and right> 
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of the error point can result in significant improvement. The proposed algorithm can 
also be used witlj multiple cameras. It will probably reduce the number of cameras that 
are necessary for successful occlusion detection. 



Fig. 8 is a block diagram of a data processing system (800), which has at least 
one processor (820)^i3sto|;age (840). Storage (840) of system (800) includes one or 
more images, comparison software (860), and data structures used by the matching 
algorithm. The steps of the dfesdriUed embodiment of the present invention are 

V 1/ \ 

performed when instructions^a compter program are performed by processor (820) 
(or another appropriafe^robessor) executing instructions in storage (840). 

System (700) alsbsipcludes a network connection (880), which connects system 

\ 

(800) to a network such a^hejntejnet, an intranet, a LAN, or a WAN. System (800) 

\ 

also includes an input qevice (845), such as a keyboard, touch-screen, mouse, or the 



like. System (800) also incluaes^ggoutput device (830) swph as a printer, display 

/ \ 

screen, or the like. System (800) al^o includes a computer readable medium input 
device (880) and a co/nputer readable medium (862). Compute\readable medium 



(862) can be any appropriate medium that has instructions, such as,those of 



comparison software (860)>^tored thereon. These interactions are loaded from 
computer readabl^ medium (862) into storage area (840). Instructions car^also be 

"m of a carrier wave over network connection. 

\ 

Thus, the instructions and data in storage (840) can be loaded into storage via aainput 



loaded into storageWea (840) in the fo 




device (880), via a network/sqch asjhe Internet, a LAN, or a WAN, or can be loadec 
from a computer readable medium such as a floppy disk, CD ROM, or other appropriate^N 



1 9 1 00/037 1 6/DOCS/988946.3 



16 



\ 

ute\ 



* 



computeXreadable medium (862). The instructions can also be downloaded in the form 
of a carrier wave over a network connection. 



System 0§pO) also includes'an operating system (not shown). A person of 
ordinary skill ipNtjie^rt wilLundefstand that the storage/memory also contains additional 
information, such as a^pcation programs, operating systems, data, etc., which are not 
shown in the figure for tKfc safe of clarity. It also will be understood that data processing 
any other data processing system described herein) can also include 

, V x 

numerous elements not sfoowin\such as additional data, software, and/or information in 

^^oardsl djsplay.devices, network connections, additional 

A 



system (800) (or/any 



memory, disk drivesr 
memory, additional^CPUs, LAN 




, input/output lines, etc. 

- j\ 

Although the de^^ptioffSb^ve coFjtains many detailed descriptions, these 
descriptions should not be s construed as limiting the scope of the invention but merely 
as providing illustrations df^tfFrieof the presently preferred implementations of this 
invention. For example, althoQsh tjals method wasxJescribed with reference to standard 



rectangular images, this method can be used to correct images of any shape or size 



Additionally, although i(ie method w^s described with reference to a particular 

j\ " \ 

correspondence method, dttjer correspondence methods could be applied including 
correlation-based matching, MRF-based matching, feature-based matching and phase- 
based matching. Thus the scope of the invention should be determined by the 
appended claims and their legal equivalents, rather than by examples given. 
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